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ABSTRACT: 

Reviewing and critically analyzing the works of others is a rewarding process in terms of one's own 
learning and development as highlighted by Wood & Kurzel (2008). Those who engage in such an 
analysis and reflective process gain more opportunities for developing their evaluative and critical 
abilities (Sluijsmans, Dochy, & Moerkerke, 1998). Identifying the good and bad approaches adopted 
by other others in their works help reviewers in their own works (Ballantyne, Hughes, & Mylonas, 
2002), as through this process they acquire a deeper understanding of the writing standards 
(Hanrahan & Isaacs, 2001) and development of a life-long learning ability (Boase - Jelinek, Parker, & 
Herrington, 2014). Professor Simon Peyton Jones, a renowned researcher at Microsoft and Glasgow 
University, also identified the process of writing as a learning activity in itself that pushes the writers 
to be clearer, more focused and being more aware of what they do not understand. 

With this aim, analysis and review of the methodology of two purposively selected journal articles 
related to pedagogy in higher education is undertaken in this paper. The scope of this review covers 
an examination of the research problem (identification, appropriateness, and explanation), literature 
review, theoretical underpinnings, methodology, research design and settings, methods and 
instruments, ethical considerations, data analysis, identification of potential limitations, conclusions, 
generalizations and reporting. 

The lessons learnt through this research review study will be beneficial for reviewers and young 
researchers in improving the quality of their own research. 

INTODUCTION 

For this analysis, the following two papers have been purposively selected based on their 
appropriateness for current research and relevant pragmatic considerations. The articles have been 
de-identified in accordance with the ethical requirements for current study (as the objective is to 
present critique of research techniques only): 

1. Article 1 deals with integration of teaching and research in a first year food and nutrition course. It 
was published in a recognized educational research journal (included in the ERA list of journals, 
2015) in the year 2012. 
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2. Article 2 deals with methods to induce research-mindedness in the students undertaking science 
education. This article was also published in a recognized educational research journal (included in 
the ERA list of journals, 2015) in the year 2010. 

These papers will be referred to as 'article 1' and 'article 2' in the rest of this research analysis paper. 
Both the articles claim to follow the 'case study' methodology on research-led education in 
undergraduate courses. The research-led education is considered to be an important study area 
among contemporary researchers (Jiang & Roberts, 2011). The research work for both the selected 
papers was conducted in the subject areas of nutrition and health sciences (article 1) and 
neurological sciences (article 2). The authors for both the articles are faculty members at the 
respective universities where these studies were conducted. The same methodological choice (case 
study), same research objective and similar contexts for the two selected research papers presented 
us with an opportunity to closely examine and analyze convergences and divergences in researchers' 
approach and quality of research. 

RESEARCH ANALYSIS: 

Following is an analysis of various methodological, design and reporting aspects of selected papers. 

Problem Definition and Research Objective 

Defining the problem that a research study is going to address is a highly important initial step in the 
research process and in its absence, there is a high probability of remaining research process to 
remain ineffective and non-productive (Zuber-Skerritt & Knight, 1986). Following is a brief analysis of 
both articles with regards to the problem being solved and the research objective. 

A proper introduction to the study is missing in the Article 1 (though the heading 'Introduction' does 
appear at the beginning of the article) and the article begins directly with a literature review (though 
there is no such heading as 'literature review'). The objective of the study, to present a case study 
that outlines curriculum design and pedagogical strategies aimed at the integration of teaching and 
research to enhance student engagement and learning, is presented on the second page in the 
article. This is based on the problem that there is a strong imperative to inspire in students the 
passion and the capacity to generate and investigate important question that impact on society, and 
that there is a need to explore approaches to achieve this within higher education. However, this 
"need" is not substantiated with evidence and it is accordingly considered a major weakness in the 
problem definition. 

In the article 2, the objective is to address the research practice at the undergraduate level by 
introducing students to the practice of research through an inquiry-based course design. Relevant 
background is provided which is gradually led to the objective. However, research problem which led 
to this study is not explained. Moreover, there are a number of specific claims made at the 
beginning of the article (no title is provided to the first section) to establish the background and 
significance of the study, however, the references are missing with most of them (as further 
explained later on in this analysis). 

LITERATURE REVIEW: 
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Dr. Dena Taylor defined four key criteria for evaluating the quality of literature review: 

1) Organized around the problem and objective of research, 

2) Synthesis and summarize what is known and what is not, 

3) Highlight any controversies and disagreements in literature, and 

4) Formulate questions that need further research (Taylor, 2008). 

The criteria proposed by Bolderston (2008) also focussed on these four aspects. The following is an 
assessment of the literature review in both the articles in line with these aspects. 

Review of both the articles suggests that main body of recent and important literature in the field 
has been referred to, as evident from the literature review presented in the articles (though the 
heading 'literature review' is not present in both the articles, and the review of literature is spread 
between various sections) and the list of references. However, there are differences in its scope, 
structure and presentation of literature in both the articles. 

Article 1, which is aimed at elaborating the 'teaching-research nexus', contains approximately one 
and a half page on a particular explanation of this concept (i.e. impact on the effectiveness of the 
teachers due to the research that they conduct). Later on, it is revealed that the definition and 
explanation discussed extensively from the very beginning of the article is not what the researchers 
intended to use in rest of the research. This appears to be disproportionate and is not in alignment 
with the literature review quality factors highlighted by Taylor (2008), Pautasso (2013) and 
Bolderston (2008) who have emphasized the importance of staying on focus during the literature 
review process. 

Article 2 was found to be relatively more focussed on the key objectives of the research. However, 
the content which could be classified as the 'literature review' was not extensive enough to inform 
the theoretical framework for the study and was confined to approximately one and half pages 
(including the beginning section which can be classified as an 'introduction' in the absence of any 
specific title given to it in the article). 

One of the positive aspects is that both the research articles show various perspectives about certain 
concepts (e.g. the ways to interconnect research and teaching as explained in a section pn 
disciplinary variation in the teaching-research nexus in article 1, and various perspectives regarding 
research-led education as presented in the first section of article 2. This approach is considered to be 
in accordance with the requirements for literature review highlighted above. 

In terms of identifying the gap in existing literature, there are no gaps explicitly identified in the 
article 1 and the apparent premise is that everything is known. This is not in alignment with one of 
the key objectives of conducting the literature review as pointed out by Levy and Ellis (2006) who 
highlighted the need for identifying the relevant gaps in literature during the review process. There 
is a statement made in the article which expresses an identified need for exploring approaches to 
achieve desired objectives within higher education; however, this is not backed up by any findings 
from the literature review. Article 2, on the other hand, does highlight some gaps in existing 
literature. One of the identified gaps in literature is the potential for confusion as to what research- 
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led education (or any related term) means. The authors of article 2 also specifically highlighted the 
body of literature that the article contributed towards (first section in the article). 

Referencing 

Both the articles have used APA referencing format. All references were found to comply with 
relevant formatting requirements as explained in the 6th edition of the Publication Manual of the 
American Psychological Association (APA). However, a gap was found with regards to the provision 
of references with specific claims. The researchers are required to present sources for their specific 
claims if they are found from the existing literature or state if they are their own opinions and 
interpretations ("Literature Reviews," 2012). We see that in both the articles, there are numerous 
statements which are not referenced and a reader cannot distinguish if these are authors' own 
opinions or findings from the literature review. For example, in one of the statements, it is 
mentioned that the effective teaching is not a process of transmission, but rather the facilitation of 
others' learning and discovery. This is an explicit claim which the authors mentioned in order to 
counter an argument from one of the previous research. This claim is unsubstantiated with the 
evidence. Similarly, article 2 has many such statements. For example, in a statement, the authors 
mention that, while the most authentic way of teaching research skills may be through guiding 
students in the conduct of their own mini-research projects (and this was included in the course to 
some extent), this is a more resource intensive activity than most courses can afford. Again, this is a 
specific claim. However, the source for this information is not provided. In article 2, from the second 
half of page 2 till the end of the article, references are very rarely found despite specific claims being 
made. This is considered to have significantly affected the research quality. 

Research Methodology 

Researchers of both the articles claim their research to be the case studies (however they have not 
specified the type of case study they intended to employ). This claim (of using the case study 
methodology) is, however, not supported by the definitions and explanations provided by the three 
key researchers who shaped the contemporary form of case study research [Yin, Stake, and Merriam 
(Yazan, 2015)]. These key researchers have explained that case studies collect data in the non- 
manipulated natural settings where researchers do not have an influence on the settings and 
contexts during the study (Merriam, 2009; Stake, 1995, 2013, Yin, 1993, 1994, 2003, 2006, 2009, 
2011, 2012, 2013). Analysing both the articles suggest that the context in both the studies under 
review is set by the researchers who have shaped the courses in certain ways, changed the variables 
through interventions and then observed the effectiveness of their interventions. 

However, if we assume that the process of implementing the changes in the courses was completed 
before the commencement of descriptive case study research and is not woven into the study 
(unlike how it appears in the articles) and hence keeping the researchers free from changing the 
variable (as events would have already happened in that case), both the papers can be classified as 
'descriptive case study' which aims to describe the situation in a narrative form (Zainal, 2007) with 
an a-theoretical perspective (Kaarbo & Beasley, 1999). 

However, if the interventions were happening as the case study was in progress, we can consider it 
closer to a less- commonly known type of case study research called 'one-shot' case study which falls 
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within the realm of 'experimental research' (Mertens, 2014). Mertens explains that this type of 
research, the researcher introduces a treatment (which in these research papers would be the 
addition of research-based components in the course) and then observes the result. There are 
certain limitations to this type of study design e.g. generation of a very weak scientific evidence, 
misplaced precision and extremely weak internal validityand so on (Hitchcock, 2016). Mertens also 
considers this to be a very weak design that does not allow for reasonable inference. She also 
explains that even if the changes are seen in the population under study, researchers cannot be sure 
about what caused them. It is important to highlight that the acceptance of this research design, 
though very weak, is linked to the collection and analysis of data that supports certain causal 
relationships. In the case of both the research papers selected for this study, this aspect is missing, 
as explained in next sub-section. 

Research Design and data collection 

From the perspective of 'One-shot case study research. Research design in both the papers was 
highly questionable from this perspective. Both the articles altered the course delivery to include 
research-based components. However, the rationale for adopting a particular approach in this 
regard was not substantiated with evidence. What informed such selection and approach remained 
unexplained. The modified course was conducted based on what researchers considered to be the 
appropriate method (without any references to established or theorized practices). There were no 
measures in place to establish the effectiveness of the approach. Both the articles acquired student 
feedback which was generic in nature and had no substantial grounds to prove that the approach 
followed in their courses was effective. 

Moreover, there is a lack of clarity in article 1 regarding what is explained in the 'method' section. 
According to this section, various research-based activities included in the course are considered to 
be the 'methods'. The activities required students to gather certain data about themselves. In the 
opinion of the analyst, this was not a suitable data for this project [in accordance with Yin's (2011, 
2013) recommendation regarding data to be linked to propositions and objectives of research]. The 
appropriate data for this project would have been the one which could provide the evidence in 
relation to the objective of the study i.e. effectiveness (or lack thereof) of the approach adopted for 
integrating research-based activities in the curriculum. No instruments were utilized, in both the 
research projects, with regards collection of such data. 

Data analysis, in both the research papers involved collection of the course evaluations by students. 
Article 2 clearly mentioned that the survey form was not tailored for the innovations in the course, 
ratings of the course coordinator's effective as a lecturer were collected. This shows a lack of 
alignment between the data and research propositions or objectives (which are considered to be 
necessary in the case study research design as highlighted by Yin, 2011, 2013). Article 2 alos presents 
subjective findings (not collected or observed in a systematic way as part of the research) such as 
the claim that the assessment results in the study gave a good representation of the students' 
scientific abilities. This (and other similar claims) is not substantiated by any qualitative or 
quantitative data collection and analysis. 

The case with article 1 is not much different from article 2. We found a number of completely non- 
related data being presented while the relevant data could not be found in the 'results and 
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discussion' section. A table has been added which summarizes the key findings with regards to the 
measures used to gauge the effectiveness of the intervention. These measures include: 'assessments 
being clear', 'teaching staff giving helpful feedback', 'workload being manageable' and 'the unit 
being well-taught'. The author of this research analysis paper failed to understand how these 
findings are related to the objective of the research i.e. effective integration of disciplinary teaching 
and learning with research. This, accordingly, is considered to be a major weakness in the research 
quality. 

From the perspective of descriptive case study. Descriptive studies employ various methods and 
instruments to gain detailed insights about the case (Yin, 2011). However, in the articles selected for 
this study, no such methods were found to be employed. The article presents a brief literature 
review followed by the researchers' recount of what was done in the courses that were modified to 
incorporate research-based elements. Specific insights about the participants (e.g. students or any 
other involved parties) were not obtained. As mentioned earlier, the student course evaluation 
survey, in both the studies, did not appear to measure the impact of changes in specific terms. 

Another Perspective. Hyett, Kenny, and Virginia Dickson-Swift (2014) did a meta-analysis of case 
study research and identified that a third of the papers which claimed to be the case studies made 
limited or no reference to the methodological aspects put forth by the principle researchers. They 
identified them as using the 'case report' method as opposed to cases study methodology. Hyett, 
Kenny, and Virginia Dickson-Swift (2014) presented three examples for such case report methods in 
their research. Their second example ('storytelling being shown as a method that can be used with 
the indigenous communities as a participatory method' which did not involve the indigenous 
community in the analysis of the case) has many parallels with the two case studies being analyzed 
in this paper. In relation to the methodological discrepancies highlighted in the section on 
'methodology' and the design aspects highlighted in this section, we can consider these studies to be 
closer to case reports. 

Ethics: 

The research participants (students enrolled in the course being analyzed) were required to 
complete an evaluation form, as reported in both the articles. They also went through a modified 
course design which can be considered as a type of experiment. Article 1 specifically mentions about 
approval from university's research ethic committee (in the section on 'methods') for conducting the 
study as well as for using data collected by students in the analysis. However, there is no mention or 
explanation about the ethical consideration or approvals in the article 2. 

Analysis and findings 

There is a clear disconnect, in both the research papers, between the data and the research 
objectives. As already explained, the objectives of introducing students to the practice of research 
through an inquiry-based course design, and the integration of teaching and research to enhance 
student engagement and learning, are not measured by any specific instruments. The analysis of 
these objectives was not found to be substantiated by the findings of student survey which, in both 
cases, which focused only on general course features. 
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We see claims for generalization in article 2. For example, it is mentioned that although the case 
study at hand was based in the natural sciences, the same approach to teaching research skills could 
easily translate into other disciplines. Though many researchers have argued that single case studies, 
particularly the descriptive type (Kaarbo & Beasley, 1999), cannot be generalized (Kennedy, 1979), 
Yin suggested that results of single case studies can be generalized to theories and when many cases 
support the same theory under same contextual factors, it can the supported theory can be 
considered to be generalized (Yin, 2003). In accordance with these requirements, a direct claim for 
generalization without a mention of other cases (that support the same finding under similar 
contexts), cannot be considered as appropriate. 

REPORTING: 

It has already been explained in this analysis that both the papers presented confusing views about 
key research elements such 'case study', 'research methods' and other aspects. It has also been 
explained that data collection tool (student evaluation survey) and its lack of ability to measure the 
results of interventions in a meaningful way are also considered a weakness. Based on these aspects, 
the overall report becomes very weak. Article 1 contains more problems with regards to 
presentation in comparison to article 2 [e.g. no introduction section, no definition objective till the 
page 2 in the research paper, grammatical mistakes e.g. "...two-thirds of students surveyed belied 
secondary school did not did not prepare them for L/niversity..." (2012, p. 1368) etc.]. 

CONCLUSION: 

This research analysis study was found to be very useful in terms of providing the analyst with an 
opportunity to deeply reflect on research methodology, design, and presentation of selected 
research papers in order to assess their credibility and improve his own critical thinking. 

Though both the papers contained some major flaws with regards to quality, article 2 was found to 
be better positioned as compared to article 1. Both the articles adopted a similar approach and were 
based on the same research objective. However, the methodology, which was claimed to be the 
'case study' was not found to be substantiated with appropriate design. This further signifies the 
point made by Yin (2013) who suggested that the awareness about the correct methodology for 
conducting the case studies need to be made more common among the case study researchers. 
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